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ABSTRACT 

Volume 3 of the report describes the Management 
Information Systeia (HIS) for monitoring instructional programs 
developed for the New York State Education Department. The background 
of the system is discussed with regard to: the specially developed 
Planning-Programing-Evaluation processes, characterized by: 
operational as opposed to philosophical objectives, strict budgetary 
considerations, effectiveness measures, resource allocation plans^ 
and action-oriented feedback; the monitoring system based on a 
continual vigilance of ongoing activity; and an experimental 
monitoring system, the Instructional Support System (ISS), developed 
for the elementary mathematics program in the Patchog,ue-Medf ord (New 
York) school system. Also covered are the seven characteristics of an 
MIS: an educational program geared to developing a comprehensive 
occupational program of wide suitability and availability; an 
educational rree illustrating the sufaunits of the general program; 
objectives, determined primarily by the program content and the 
problems encountered in managing a particular program; modules, and 
the adaptability of these smaller instructional units in the 
traditional course; measurements, and the superior suitability of 
criterion-referenced tests over norm-referenced tests; system design, 
especially regarding the need for computer assistance; and 
management, especially in the diagnosis and feedback functions. 
(JR) 
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I. BACKGROUND 

A. Planning Progranmiing Evaluation (PPE) 

During the past several years Riverside Research 
Institute CRRI) has developed a series of processes called 
Planning-Proqranuning-Evaluation (PPE) under contract (s) with 
the State Education Department (SED) . PPE can be traced to 
the Planning-Programming-Budgeting (PPB) Systems which were 
initially introduced in the Department of Defense, and sub- 
sequently extended to other federal and state agencies to 
improve the management of government programs. Whereas PPB 
assumes that a classical hierarchical organization exists 
with appropriate Management Information Systems (MIS) incor- 
porated at the lowest levels of the structure, PPE does not 
make this assumption. In fact, the design of such a MIS for 
monitoring instructional programs is the focus of the present 
effort. Since the conceptual model designed for monitoring 
instructional programs is a PPE subsystem, a brief overview 
of five essential PPE concepts will be presented. 
1. Operationalized objectives 

The PPE System does not accept broad, philosophic 
program ob:Iectives. It requires the consideration of alterna 
tive objectives, with continual awareness of the realities of 
limited resources. Upon reaching general consensus, the 
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accepted program objectives are subjected to a refinement pro-- 
cess directed toward the attainment of a level of specificity 
which will permit an analytical evaluation of progress. This 
process involves listing alternative procedures which could 
be employed to determine the deqree to which the objectives 
have been met. The selected procedures (or operations) represent 
the operational definition of the program objectives, and are 
referred to as "operationalized objectives". Therefore, as 
ooDosed to setting a general objective such as: "Students 
should show improvement in elementary language arts"^ one might 

establish an operationalized objective such as: " students 

will be able to write effective English as demonstrated by 

performance on tasks (or tests)". The learning time and 

evaluation period would also be specified, as well as any methods 

of comparison (e.g., improve amount in comparison to whom or 

what) . The statement of objectives in operational terms out- 
lines the indicators to be used for program effectiveness, and, 
by doing so, requires the further consideration and explication 
of concrete goals. 

2. Objectives - activity - budget linkage 

It is not unusual to find management systems wherein 
objectives are established in isolation, without regard to 
budgetary concerns. Goals are often divorced from the realities 
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of operations. In the PPE System, however, objectives are 
closely linked with plans for achieving the objectives. The 
PPE "progian structure" identifies the significant programs 
within the system. Objectives are specified for each program. 
Activities and their corresponding costs are associated with 
the program objectives. The program structure includes further 
subdivisions to subprograms, sub- subprograms, etc. Each level 
requires a more concrete identification of objectives, opera- 
tions and costs. 

This linkage of objectives, activities and budgets 
through a program structure allows one to pinpoint allocated 
resources in relation to program objectives. It facilitates 
comparisons of alternative methods and provides an effective 
management tool. 

3. Effectiveness measures 

The program structure informs organizational members 
of the objectives which they are expected to achieve. It also 
provides information on the means of achieving them and the 
resources available. Perhaps of greater import to the in- 
dividual members, the structure specifies the manner in which 
progress will be measured. The PPE System focuses significant 
attention on the measurement system. This emphasis is well 
justified in view of the usual weight that evaluation bears 
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upon the motivational substructure within organizations. Apart 
from such concerns, however, PPE leans heavily upon the measure- 
ment systems as a basis for self-corrective decisions during 
the course of program implementation. 

As previously noted, part of the process of estab- 
lishing objectives requires the selection of procedures to be 
used to determine the degree to which the objectives have been 
met (i.e., "operationalizing the objectives;" see I.A.I.) . 
Operationalized objectives provide a basic source for indicators 
of program effectiveness. These indicators can be applied at 
various points in time (e.g., mid-program, terminal point, etc.) 
to evaluate effectiveness. Yet, a dynamic management system 
requires continual measurement of progress with related cor- 
rective mechanisms available for application at the earliest 
point in time. For a manager to steer such a dynamic system, 
a management information system must be provided which will 
yield reliable, valid, and timely information. Of equal im- 
portance, the information must be closely tied to corrective 
actions which exist within the manager's repertoire. Informa- 
tion without action-potential is expensive, inefficient — and 
frustrating. A defective thermostat which only registers the 
temperature merely tells you what you don't have to be told — 
if you are cold. A PPE measurement system will not only 
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diagnose progress, it will also uncover problems; and it will 
prescribe appropriate actions to adjust an ongoing program. 
As noted, the corrective actions must be cast within the 
bounds of resources available to the manager. Figure 1 
illustrates this measurement system in contrast to the usual 
system. 

4. Resource allocation plans 

In order to effectively manage a program with 
established objectives, one must develop a detailed plan where- 
in resources are allocated to the various components of the 
program. Whereas, line-item budgeting is oriented toward re- 
lating costs to resources, the PPE System focuses upon costing 
programs. Beginning with program objectives, the alternative 
means of accomplishing these objectives are considered. This 
comparison of opt.lons is essentially the core process involved 
in developing resource allocation plans. 

Resources include all assets. However, the relative 
dollar value of such fixed assets as physical plant facility 
should not be the primary basis for determining degree of 
emphasis in planning. Variable assets, especially manpower, 
should be central in view of the annual budget portion occupied. 

Resource allocation plans are developed with the 
aim of utilizing one's assets in the most cost-effective 
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method to achieve program goals. They are appropriate at all 
levels in a program (e.g., state, regional, local, buildinq, 
classroom); rheir specificity will vary with levels. At the 
higher and more general levels, PPE Systems permit more 
realistic financial planning. The program structure with 
definable units allows one to more accurately predetermine 
costs than was previously possible using line-item budgeting 
and retrieval accounting procedures. At the more detailed 
levels, PPE Systems require resource allocation plans which 
identify the student and teacher types; class and faculty 
organizational structure and size; instructional materials; 
physical resources and equipment; space; and other variables. 
Alternatives available include various class groupings; team- 
teaching models; master teachers; etc. Within a classroom, 
teachers are concerned with the optimal utilization of their 
time. Instructional methods and materials, as well as class 
groupings, can significantly influence the effective use of 
this major oersonnel cost. 

5. Action - oriented feedback 

In discussing effectiveness measures, it was pointed 
out that information without action-potential is of little 
value. E'igure 1 showed the PPE feedback function affecting 
the instructional activities. The same measurement system can 
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be eiriployed to assess and prescribe the optimal allocation of 
resources within an ongoing program. By evaluating the results 
of having allocated resources and having undertaken a unit of 
instruction, one can determine the degree to which objectives 
were achieved. Consistent with a dynamic management system, 
one can diagnose deficiencies and prescribe appropriate adjust- 
ments if necessary. 

Thus the PPE System consists of two feedback loops, 
one affecting resource allocations, the..other affecting 
instructional activities. (See Fig. 2). From among the array 
of alternative methods of allocating resources, one must 
actually take action (i.e., "Decision-taking;" see Figure 2). 
In a similar fashion, one must also select actual instructional 
activities. As a result of these decisions, the learning 
sequence is structured. Finally, student performance is 
measured and related to resources, activities, etc. In con- 
trast to the traditional use of such measures, however, an 
action-oriented MIS interprets the information for diagnostic 
purposes and feeds it back to the ongoing program. (See 
"Diagnosis and Feedback Function", Fig. 2). Thus the loop 
between performance and action is closed. 

In addition to yielding managerial information for 
the current operational program, the MIS can also provide 
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inputs for an evaluation system. Such systems are often de- 
signed to measure performance in terms of program effectiveness. 
Thus MIS data can be reinterpreted to yield evaluation infor- 
mation permitting feedback, on a longer term basis, to banked 
resources and instructional activities (see Figure 2) , 
B . Monitoring 

Following the planning stage, a program should be 
implemented in conjunction with a monitoring system which is 
capable of: assessing whether the implementation is in accord 
with the original design; measuring the degree to which it is 
meeting its objectives; providing the management necessary to 
detect problems and effect operational adjustments. 

The two feedback loops discussed in the previous section 
can be actualized at the local level through a program monitor 
or manager. Information provided from the measurement program 
can serve as a primary source for a management information 
system which enables the monitor to make decisions regarding 
necessary adjustments in the instructional program. 

The key concept underlying a monitoring system is con- 
tinual vigilance of ongoing activity. As compared with an 
evaluation system, monitoring involves tracking: 

1. Student performance 

2. Instructional activities 
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3 . Resources 

(a) People 

(b) Materials 

This continuous check on program elements, which is 
designed to ensure measured progress toward objectives, 
differs from evaluation systems which are intended to provide 
assessment data at a higher management level. Evaluation 
systems frequently assess performance, not only in terms of 
local objectives, but also focus on external goals. In 
addition, they often depend upon norm-referenced measures 
(e.g., typical standardized achievement tests). Even when 
they depart from such measures, they tend to use comparative 
base for evaluation. For example, one might tabulate the 
number of education-related jobs received by the graduates 
of a particular program. By collecting similar data through- 
out the state, a data base is assembled which permits the 
evaluation of programs in relation to statewide performance. 
There is no doubt, evaluation systems serve critical functions, 
but functions which differ from those served by monitoring 
systems. However, the data garnered om a monitoring program 
can provide input for an evaluation system. Even though a 
monitoring program will rely heavily upon criterion-referenced 
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measures,* which require an absolute as opposed to a compar- 
ative basis for evaluation, these data can be reinterpreted 
in a norm-referenced manner. For instance, a monitoring system 
could determine the number of students who mastered a specified 
set of instructional objectives within a unit o£ time through 
the use of identified resources. For monitoring purposes, one 
might relate this information, which details what actually 
occurred, to the planned or intended performance. A.ny dis- 
crepancies ^^70uld be identified and corrective actions would be 
taken. Yet, this same information could provide a valuable 
input to an evaluation system designed to make periodic assess- 
ments of the performance of many similar programs. 
C. Instructional Support System (ISS) 

In ]969, the Patchogue-Medf ord (New York) school 
system, with the assistance of RRI, initiated an experimental 
monitoring system on a limited basis. 

The elementary mathematics program was chosen as the 
area in which to conduct the first monitoring efforts. This 
\'7as due to the local concern regarding recent indicators of 
deficiencies in mathematics achievement. The first step re- 
quired the development of detailed instructional objectives 

See Section II E for a further explanation of criterion- 
referenced measures. 
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for the mathematics curriculum. Mastery tests and "drop-tests 
(i.e., centrally administered tests with content unknown by 
teachers) were developed and incorporated into the program. 
A major goal of this limited trial was to determine the 
utility of alternative reporting modes. 

In addition to the many lessons typically learned 
through an actual operational test in a real district, the 
project yielded some evidence for the utility of a flexible 
resource allocation system. On the basis of measurement in- 
puts from the monitoring program, the district noted a dis- 
crepancy between the intended and actual performance. This 
signaled the allocation of supplementary math resources to 
those buildings identified through the tracking system. Con- 
tinued vigilance of the program indicated that the limited 
application of a flexible resource allocation system appeared 
to yield positive, if modest, benefits to the mathematics 
program. 

RRI's experience at Patchogue and other districts re- 
sulted in the tentative conclusion that application of the PPE 
model to educational programs at the local level appeared 
feasible, and possibly could yield significant results. 

As this field experience was being acquired, RRI began 
to translate design concepts into an instructional monitoring 
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system v/hich would be sufficiently elaborated to allow for 
further experimentation. As these advances were taking place , 
RRI was benefiting from similar work at SED in Occ. Ed. By 
keeping abreast of the developments in Occ. Ed., and by gaining 
field experience in an Occ. Ed. local environment through the 
test-bed work at Nassau, the initial outline of a work plan 
for test-bedding a system in Ojc. Ed. has been developed. As 
further progress is made in establishing instructional objec- 
tives and in the modularisation of instruction in Occ. Ed./ 
the feasibility of actually test-bedding an instructional 
monitoring system will be explored. Information regarding the 
relative value of alternative approaches will be furnished as 
a result of simultaneous work being conducted in the first 
major attempt to install such a system in a local school dis- 
trict. 

In August, 1971, the Guilderland Central School Dis- 
trict (GCS) and RRI embarked on the development of an 
Instructional Support System (ISS) for the elementary mathe- 
matics and reading programs.* This three-year project is 

* ■ 

Since the monitoring program was designed to support the 
implementation of instructional programs the district coined 
the term "Instructional Support System" (ISS) . 

The term was chosen with the aims of emphasi7.i r^g the teacher- 
support aspects of the system and minimizing any pos.*ibility 
of repercussions from the negative connotations associated with 
monitoring and managing concepts. The utility of this term in 
meeting these aims has not been fully tested. 
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aimed at the installation of an instructional monitoring pro- 
gram which can be locally supported and transferred to other 
districts. It represents the first attempt to establish an 
operational system which is managed by a permanent, full-time 
program monitor. The experience which will be acquired from 
this project, as well as the software packages, should reap 
significant benefits to development of an MIS in Occ. Ed. 

II. Characteristics of an MIS 
A. An Educational Program 

The purpose of occupational education in New York 
State is described in the Long Range Program Plan in terms of 
eight broad goals. The first of which is stated as: 



developing a comprehensive occupational 
education program, suited to individual 
needs, interests and abilities, and 
available to persons of all ages and in 
all areas of the State. 



The overall Occ. Ed. program is categorized into seven 



major content areas: Agriculture, Distribution, Health, Home 
Economics , Office, Technical, Trade/Industrial . These major 
content areas are each subdivided into a number of specific 
terminal programs representing differing specialties. For 
instance, the health area includes ten program?; (e.g,, dental, 
laboratory, etc.). Consistent with the stated goalrw a 
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comprehensive occupational program has been developed in New 
York State. It includes several subprograms, which can be 
further divided into sub-subprograms. In the interest of 
clarity, however, any defined program, to which one can assign 
terminal objectives and apply an appropriate, unique title, 
will be referred to as an educational program. Thus one can 
reference the broad program of occupational education or a 
more specific program aimed at the development of specialized 
laboratory technicians. 

All fully developed programs are based upon a plan 
for the allocation of resources. The program details the 
relationship between its goals and the means required to 
achieve the goals. A specific educational program will begin 
with intended objectives and proposed measures of effective- 
ness. These will then be related to the activities which will 
be undertaken as the means by which the objectives are to be 
met. Whereas program objectives describe intent, program 
activities specify the way in which the intent is to be ful- 
filled. The fully documented plans should outline goals and 
means, and measures of inputs and outputs all of which are set 
within a specified period of time. 

General educational objectives will be common across 
specific programs. A taxonomy of objectives for broad domains, 
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such as cognitive, affective, psychomotor, might be appro- 
priately applied. The terminal objectives of a specific 
educational program can be related to the general taxonomy. 
By examining such a crosswalk, one might uncover aspects of 
a program which require further attention, or one might develop 
insights regarding the relationships among program elements 
or across different programs. For example, the psychomotor 
skills required in meeting certain objectives in a radiology 
program might be quite similar in a nursing program. This 
could lead to re-clustering portions of the program. 
B. An Educational Tree 

As noted in the previous section, the general occupa- 
tional education program can be subdivided into many subunits . 
A picture of these subdivisions linked with elementary educa- 
tion yields an "educational tree" with a common trunk and many 
branches (see Figure 3) . Although references to the numerous 
subdivisions , their parts and subcategories can , at times , 
cause semantic confusions, the picture is worth viewing. It 
can yield insights and suggest viable alternatives to current 
organization. 

Occ. Ed. has embarked on a project aimed at developing 
modular instruction. By identifying those objectives which 
are common to various portions of a total program, they can be 
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reorganized into common clusters and presented as an instruc- 
tional package (see Figure 3) . This not only improves 
efficiency/ but also permits greater flexibility for students. 
Consequently, a student can take a unit of instruction which 
will be beneficial to his development in several terminal 
directions. This allows him more time in deciding upon a 
specialty. It also reduces the amount of re-cycling one must 
undergo in order to change specialties later. 

These common clusters can be organized in various 
ways. Related objectives could be set into the traditional 
course or program organization which usually spans a half year 
or a year. An alternative would be the organization of objec- 
tives into instructional units of shorter duration. One 
common unit has been termed a "module*' which typically is a 
two-week period of instruction on an integrated set of objec- 
tives. A combination of the traditional course and tne module 
unit can be achieved by organizing modules into course lengths. 

One advantage of the module unit is that it affords 
the opportunity to build "mini-trees" within a program. A 
main track sequence of objectives could have module branches 
for compensatory or enrichment purposes. There are numerous 
ways of arranging the elements. For instance, as opposed to 
requiring a transfer student to re--cycle, one could develop 
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"crosswalk modules" which include just the material not 
included in the original program (see Figure 3) . 
C. Objectives 

Objectives have been referred to in several previous 
sections (e.g., lAl, IIA, IIC) . It has been noted that 
objectives should be operationalized so as to permit an 
analytical evaluation of progress. Yet, as educational pro- 
grams vary in comprehensiveness, from the overall occupational 
education program to the laboratory technician program, program 
objectives will vary in a similar manner. Although vague 
statements regarding desired outcomes are always to be avoided, 
the level of specificity and detail required in setting objec- 
tives is determined primarily by the program content and the 
problems encountered in managing the particular program. How- 
ever, in view of the many terms currently in use, a general 
review of basic concepts might yield guidelines for setting 
objectives. 

At least ten qualifying terms have been used to define 
various types of objectives: 



1. 



Educational 



2. 



Program 



3. 



Instructional 



4. 



Terminal 
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5. Approximations-to-Terminal 

6. Course 

7. Instrumental 

8. Enabling 

9 . Behavioral 
10. Generic 

The term "educational objectives" has typically been 
used to refer to general classification schemes which list 
various types of learning (e.g., Gagne^s learning hierarchy). 
Most of these classifications were developed with the intent 
of universal application. For instance, one would expect to 
find Gagne's "verbal association" learning or "problem-solving" 
in any educational program. Bloom's taxonomy of educational 
objectives was developed with a similar intent. 

"Program objectives" are similar to the goals estab- 
lished by an administrative unit. They could be set at the 
SED level, regional or local levels. In contrast to general 
goals, they are stated in terms of the specific outcomes 
expected from the program. Using operational definitions, 
program objectives state the learner behavior which is expected 
to result as a consequence of the program. Although program 
objectives focus primarily on the instructional results, they 
can also be set for non-instructional aspects of the program 
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such as student welfare, pupil personnel and services. Thus, 
"instructional objectives" refer to those program objectives 
which are directly related to instructional activities. 

A "terminal objective" is established from an analysis 
of the specific domain (e.g*, knowledge, performance, etc.) 
that is to be learned by the conclusion of the program. For 
instance, a concrete terminal objective might be: learners 
will be able to use appropriate tools in repairing an engine. 
"Approximations-to-terminal objectives" refer to objectives 
which are established as short term or intermediate steps to 
be taken on the path toward the achievement of terminal objec- 
tives. They might be set within a time frame (e.g., a "course 
objective") to allow monitoring of progress. An example might 
be: learners will be able to use a certain tool to repair a 
certain problem with a type of engine. 

"Instrumental objectives" (or "enabling objectives") 
are often used to refer to course objectives, too. However, 
they are sometimes differentiated from approximations-to- 
terminal objectives in that the student might be required to 
learn something which aids in the development of ihe terminal 
performance, but which in itself is not required to meet the 
terminal objective. For example, students might be taught a 
special vocabulary which is useful for later classroom 
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communication as one works toward the terminal objectives 
(e.g., phonetic terminology, nomenclatures of engines, etc.). 

The term "behavioral objective" has been widely used, 
with the consequence that various denotations and connotations 
are presently associated with it. While attempting to re- 
focus attention away from setting objectives in terms of 
teacher activity (e.g., to demonstrate how to use equipment), 
many advocates of behavioral objectives stressed the importance 
of learning outcomes. The question was asked: "What should 
learners be able to do as a result of teaching?" Programmed 
instruction led others to advocate that objectives, instruc- 
tion, and testing all be on the same level. This approach 
can lead to an enormous catalog of very specific objectives; 
at its extreme, it ^^^ould require an objective for every possible 
test item. 

The level at which an objective is stated is also at 
issue when the term "generic objective" is used. A recently 
developed reading program at SED has used this term to apply 
to objectives which are stated in a form that is sufficiently 
general to permit use at different levels within a program. 
For example, a generic objective might be: "Given two stories 
orally, the student says their similarities." The intent of 
this general statement is to avoid the specificity of the 
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difficulty level of the stimuli so that the same objective 
can be used from the very low to the higher grades • 

In addition to the ten types of objectives which have 
been introduced, there are many others. The benefits to be 
derived from the various terms is not yet clear. Hopefully, 
an open approach will permit the value of the various typologie 
logical distinctions, and terms to be tested in practice. For 
the present, however, it might be of greater import to limit 
the vocabulary. This can be done without forgetting the 
reasons for the coining of the sundry terms. As a first step 
in this direction, it might be well to focus initially upon 
the establishment of objectives in a manner which m.eets the 
requirements of operational definitions, while using the more 
neutral terms (i.e., program objectives, terminal objectives, 
and course objectives) . 
D. Modules 

In the previous discussion of an educational tree 
(see Section IIB) , it was pointed out that instructional 
activities can be set into clusters which are common to various 
branches (or specialties) of the tree. These clusters can be 
organized into the traditional course length, or divided into 
smaller units referred to as "modules" or can be organized in 
a manner which combines both units. That is, common activities 
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can be organized into modules, which are then set into the 
traditional course. It was noted that an advantage of the 
module unit is its inherent adaptability, due to size, which 
permits the use, and encourages the development, of branch- 
like modules within the usual course. As a result, one would 
expect to find crosswalk modules to other programs, remedial 
modules within a program, as well as enrichment modules to 
aid the advanced student. 

An essential aspect of the module concept is the 
small unit. It has been typically set for a two week interval 
of time. Ho^^7ever, there is nothing sacred about the two week 
period. Depending upon the needs of a specific program, it 
could be set for any unit of time. It could even be a variable 
period, with some modules spanning four weeks and others one 
week. As lonq as objectives can be organized into a small 
manageable unit, with a relatively consistent time period 
linked to it, and with identifiable beginning and termination 
points, one can utilize the module concept. 

The short interval notion is based upon the need to 
identify problems at an early stage. In addition, by using 
a small unit, one can easily develop additional modules for 
special purposes (e.g., remediation). If a measurement pro- 
gram is integrated into a module system, it will permit 
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end-of-module testing and the subsequent establishment of 
decision rules which specify corrective actions should the 
actual performance be different than the performance expected. 
The short time interval allows students to be re-cycled 
through a module or to take a special module without fear of 
losing pace with others in the program. Decision rules might 
call for a re-allocation of resources. Teachers or students 
might be re-grouped. 

In addition to these advantages, the module unit is 
a core element in the development of a management system. When 
an educational program is rigidly organized into term courses 
or year-long programs, one has little flexibility, and is, 
thus, highly constrained in exercising effective management. 
By organizing a program into manageable units, one is afforded 
the opportunity to make adjustments during the course of an 
operational program. 

Besides providing a central element for a manageable 
system, the module unit can also yield the basis for financial 
planning. By attaching cost data to a module, one can identify 
the input for a smaller portion of the total program. Output 
can be easily measured by focusing on the objectives measured 
within the module. A small unit which can be costed in 
advance should be amenable to direct transfer to industrial 
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\vork programs and other similar programs which are not under 
the operational control of BOCES or districts. 

In designing the content matter to be included in 
modules, the first step should be Job Family behavioral 
analysis of the various fields to be served by the overall 
occupational education program. Detailed job analyses should 
identify the rfjquired skills in each job. After setting objec 
tives, jobs should organized into skill families. Care 
should be taken to ensure that objectives are job-relevant. 

A plan should be developed for the allocation of all 
resources which are directed toward the attainment of clearly 
stated objectives within the given period of time established 
for the module. Initial decisions regarding the allocation of 
resources are made by considering all available alternatives. 
Resources can be categorized in various ways. One might begin 
by considering the combinations possible within and among the 
following categories: human time (teacher, paraprof essional , 
learner), equipment, space, textbooks, consumable materials, 
and overhead. 

In a similar fashion, the actual instructional 
activities which will be required to meet the objectives, are 
outlined. A detailed instructional activity might identify: 
instructional method, teacher-student roles, teacher-student 

F/142-5-00 - 27 

ERiC '^'S 



characteristics, etc. Fully developed modules of instruction 
will include detailed instructional activities, as well as the 
specification of objectives, resources, time, and performance 
measures. The latter will be discussed in the next section. 
E. Measures 

An integral part of the implementation of an educa- 
tional program is the establishment of an operational measure- 
ment system. Student performance measures are central to the 
system. The total system, however, should also include data 
inputs which result from tracking instructional activities and 
resources. The overall monitoring system should provide 
sufficient information to permit ongoing managerial adjustments, 
while at the same time, yielding the data base for measuring 
the degree to v;hich the program is meeting its objectives. 

In developing learner performance measures to be used 
in a monitoring system, one cannot depend upon the norm- 
referenced testing model which serves as the basis for most 
standardized achievement tests. Although the norm- referenced 
model can be useful for many purposes, it was not developed 
with the problems of program monitoring or evaluation in mind. 
Relying heavily upon methods developed to measure abstract 
psychological constructs, norm-referenced tests are potentially 
dysfunctional when used in monitoring systems. 
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Since test specialists are usually governed by the 
general purpose of developing measures which will identify 
differences among individuals, they often select test items 
on the basis of whether they can differentiate among people. 
The technology is deeply rooted in this philosophy. Item 
analyses search for discriminability ; rules of thumb call for 
few very easy or very difficult questions. The performance 
measures which result from these tests often mask both the 
merit of educational programs and measured progress toward 
goals . 

Most norm-referenced tests assume a compensatory model 
for the derivation of individual scores. Other things being 
equal, it assumes that a student who misses all of the questions 
on engine nomenclature and does well on problem-solving is the 
same as the student who misses the problem-solving questions 
and does well on engine nomenclature. Apart from the correct- 
ness of the assumption, the usefulness of such summary test data 
to the user is questionable. The lack of specificity in output 
diminishes the range of alternative corrective actions available. 
With a potpourri of stimuli yielding responses summed in a 
compensatory manner, one simply cannot identify the deficiency 
nor begin to take corrective action* 

Even if the typical norm-ref erenced test yielded mul- 
tiple subscores, one would still not be able to determine what 
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a student had learned, or the actual portion of a content domain 
which had been mastered. This is a natural conser ence of the 
testing model which is aimed at measuring differences among 
individuals . 

A criterion-referenced test is defined as one that is 
constructed to yield scores which can be directly interpreted 
in terms of performance standards. This should not be confused 
with an achievement test which has a set passing point. To 
properly construct a criterion-referenced measure one must de- 
fine a class of tasks which one might expect an individual to 
successfully complete as a result of instructional activities. 
The test is deliberately constructed with items which sample 
the class of tasks. Individual scores relate to the tasks or 
class of tasks, not to other scores. 

Although a criterion-referenced test could look very 
much like a norm-referenced test, it differs in: purpose, 
construction, scoring, interpretation, utility. It is not 
intended to yield individual difference comparisons; it does 
not follow the same construction pattern (e.g. , moderate 
difficulty levels, high discriminability) ; the scoring system 
is not as dependent on the compensatory model (i.e., scores 
on apples and oranges are not pooled) ; it is more amenable to 
the establishment of meaningful passing points and one can 
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better determine the actual material mastered; it has the po- 
tential for far greater utility as a monitoring instrument. 
In brief, the most significant differentiating characteristics 
from the typical norm-referenced test is the absolute basis 
for interpretation as opposed to the comparative basis built 
into the norm-referenced test. 

It is expected that a monitoring system which is 
based upon criterion-referenced measures will yield improve- 
ments in the certification process. The establishment of 
passing points — a central concern for certification purposes — 
represents a critical area of neglect by traditional psycho- 
metricians. However, the absolute interpretation of data, 
through criterion-referenced measures, yields information which 
is closely linked to the actual terminal behavior. As a result 
passing points can be set in a manner which directly relates to 
the behavior sought. 

Criterion-referenced measures can be administered to 
each student at the completion of each module of instruction. 
This system would yield the means for establishing decision 
rules regarding the next unit of instruction for each student. 
It should be noted, however, that for monitoring purposes alone 
a sampling procedure could be applied ^^7hereby tests would be 
administered to only a portion of the learners. 
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By testing at the conclusion of a module^ a measure 
of effectiveness is linked with a manageable unit (i.e.^ the 
module) . A reporting system can also be established to track 
the instructional activities undertaken for each module. By 
testing on instrumental objectives, which are imbedded in 
instructional activities^ one can check on the use of the 
activities. Resources can also be tracked through a reporting 
system. Should local constraints hinder the establishment of 
a reporting system to track human time, gross measures can be 
derived from the intervals between end-of-nodule tests. Thus, 
continual vigilance can be maintained on the program elements. 
The same information can provide the ingredients for a data 
base which can be used for longitudinal studies and for input 
to evaluation systems. 
F. System Design 

It is not expected that any one procedure for install- 
ing a monitoring system will be appropriate for any large number 
of local programs. However, it is expected that the basic 
elements will be similar enough both to permit gains from ex- 
perience and to yield data inputs to a common evaluation system. 

Information processing is a major concern in designing 
the operational system. The system must be designed so as to 
allow the rapid and orderly storage and retrieval of information 
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Although such a system need not be computer-based, it is likely 
that electronic data processing will eventually be required. 
The need for computer assistance will become more apparent as 
the data base permits the application c£ sophisticatea analytical 
approaches. That is, as monitoring systems become a reality, 
and when they are closely integrated with ongoing educational 
programs, data bases will be established which will allow and 
require the application of advanced mathematical and logical 
systems capabilities to the delineation and interpretation of 
alternative decision rules and resource allocation plans. 

Even at a primitive stage, significant design decisions 
must be made. Test development procedures must be established 
with a concern for test security and the equivalence of 
measures. The design of test reproduction, ordering, admin- 
istration, and processing must be accomplished within the 
constraints and limitations of the current operational system. 

Forms must be designed for the instructional activity 
and resource reporting systems. These, especially, must be 
developed in close cooperation with the existing management 
system. In addition, output formats for reports to teachers, 
learners, parents, and administrators must be designed. 

RRI has developed computer software to match the hard-- 
ware configuration of the Albany BOCES . The system, designed 

F/142-5-00 - 33 - 

ERIC 



ERIC 



for the Guilderland School District, included relatively per- 
manent files on district schools, teachers, students, test 
answer keys, performance criterion levels for instructional 
objectives. It is capable of receiving answer sheets from 
ever six hundred different tests, as well as reports on instruc- 
tional activities and allocated resources. The operational 
system scores the tests, compares performance on each objective 
with pre-established criterion levels, and generates multiple 
report forms. 

Reports currently display for each teacher, within 
each school, the performance of students within each module 
of instruction. Presented next to each student's name are 
the instructional objectives included in the module together 
with an indication of whether the stxident was successful on 
the respective objectives (i.e., whether the criterion level 
was reached) . It also notes the individual scores on each 
objective and the related criterion level. In addition the 
dominant instructional activity used in teaching the objective 
is linked with each objective for each student. The activities 
are categorized by instructional method, material resources, 
and class size. 

Although the Guilderland experience has indicated that 
the implementation of such a monitoring system is possible. 
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there is no reason to believe that the particular configura- 
tion is optimal for all situations. As was previously pointed 
out, it is not expected that any one design will be appropriate 
for all local programs. Nevertheless, the basic elements, 
both conceptual and operational, should be relatively universal. 
G. Management 

The management of an educational enterprise is no 
simple task. It is, in fact, one of the most complex and 
demanding administrative undertakings. Local educational pro- 
grams are presently managed through the sundry informal 
organizational structures spread throughout the state, even 
though consistent formal organizational patterns exist. The 
central manager, a superintendent, must be responsive to the 
directions and policies of a board, as well as to the general 
public, which exerciser varying degrees of social and economic 
influence over the educational program. Whether the superin- 
tendent governs a local district or a BOCES , he must also 
adhere to federal and state requirements. At the operational 
level, one often finds an assistant superintendent for instruc- 
tion, who is the typical line manager of local educational 
programs. However, his authority is diminished by the presence 
both of subject matter experts (district coordinators, depart- 
ment chairmen, and teachers) and of independent administrators 
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(principals and supervisors) • Another significant force is 
the existence of the co-equal position — assistant superinten- 
dent for business, who monitors the resources. 

Once an educational program has been fully developed 
with the delineation of essential components as previously 
discussed, a monitoring system can be designed and installed. 
However, to actually implement the monitoring function, a 
program manager must be designated to make decisions regarding 
the changes in elements of the instructional program. Since 
the central concepts of monitoring include resource allocation 
and alterations in instructional activities, a role must be 
established for the execution of these functions. 

The data base which grows from the operational system 
contains the potential for an information overload. Therefore, 
a crucial aspect of the system is to design methods which will 
manipulate the information subset so as to produce reports 
which are meaningful to a manager. Meaningful managerial 
reports are those which permit operational decisions and 
actions. Information selection, methodological developments 
and report designs, all must stem from a thorough analysis 
of the repertoire of the manager. 

An actual educational program is implemented by 
initially making decisions with regard to the allocation of 
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resources and the selection of instructional activities. 
Following the learning sequence, performance measures are 
related to resources and activities. This information is 
interpreted for diagnostic purposes and is fed back to the 
ongoing program. It is here, at the diagnosis and feedback 
function, where the managerial role is central. 

To properly build an effective management information 
system to assist in this function, one must catalog the 
diagnoses possible. This should be accomplished in con^ 
currence with a list of corrective actions. The major challenge 
is to transform the data inputs into an output form which 
correlates with the possible diagnoses. A related effort must 
be taken to develop the decision machinery which will permit 
one to make probalistic judgments as to whether a particular 
output matches a diagnostic category. 

The implementation of corrective actions can occur in 
various areas and can be accomplished through a variety of 
managerial styles. Program adjustments could be made in sub- 
ject matter covered, instructional method, grouping, etc. 
Alternative resource allocation plans could be set into 
operation. The corrective action could be taken simply by 
providing feedback to students. Alternatively, the program 
manager could interact with teachers. The interactive process 
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could take various forms . 

In summary, the design and implementation of an effec- 
tive monitoring system requires a thorough analysis of the 
existing operational system. The +*.echnological innovations 
must be introduced in relation to what exists and to what is 
possible. A monitoring system cannot operate in a vacuum. 
\Vhen introdr.ced within the adaptation range of an existing 
system and within the bounds of what is possible / it can offer 
real benefits to the management of education. 
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